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Average  mutual  information  is  studied  for  nonGaussian 
measures  of  two  types:  spherically  invariant  and  Gaussian 
mixtures  (a  generalization  of  spherically-invariant) . 

A  complete  answer  is  obtained  for  finite  mixtures.  These 
results  are  applicable  to  communication  channels  perturbed 
by  nonGaussian  noise  processes  described  by  these  measures. 


Ob  amragi  mutual  information  for  spharkalljr  tanrial 
Buaivif  and  ionu  other  mixtures  of  Gautslan  measures 

1.  Introduction 

The  actual  transmission  capacity  of  a  given  channel  is  a  parameter  of  ba¬ 
sic  importance  in  any  communication  system  since  it  limits  the  rate  at  which 
information  can  be  transmitted  reliably.  There  has  thus  been  an  effort  which 
started  with  Shannon  (1048)  to  compute  the  capacity  of  transmission  for  dif¬ 
ferent  channel  and  transmission  models.  In  the  ease  of  a  continuous  channel, 
most  results  have  been  obtained  for  a  Gaussian  noise  (Baker  (1078);  Hitsuda 
and  Ihara  (1078);  Kadota,  Zakai  and  Ziv  (1071)).  Some  attempts  to  steer  away 
from  the  Gaussian  case  have  also  been  made  (Gualtierotti  (1080))  and  these 
indicate  that  new  methods  may  be  required.  Indeed,  in  the  Gaussian  case, 
most  quantities  of  interest  can  be  explicitely  obtained,  whereas  these  com¬ 
putations  are  almost  always  impossible  in  other  instances.  Furthermore  the 
computation  of  mutual  information  requires  that  the  joint  law  of  the  message 
and  the  received  signal  be  absolutely  continuous  with  respect  to  the  product  of 
the  marginals  and  that  the  Radon-Nikodym  derivative  be  computed  :  though 
the  Gaussian  case  is  well  known  (Baker  (1078)),  this  knowledge  is  again  un¬ 
available  for  most  other  models. 

Spherically  invariant  distributions,  when  the  mean  is  zero,  are  mixtures  of 
Gaussian  ones  and  through  mixing  a  number  of  well  known  distributions  can 
be  obtained,  such  as  the  double  exponential  and  the  Student  distributions 
(Keilson  and  Steutel,  (1074)).  There  is  also  some  evidence  that  some  real  life 
noises  can  be  described  through  spherically  invariant  probabilities,  particu¬ 
larly  in  underwater  acoustics.  It  is  thus  natural  to  investigate  the  problems 
of  absolute  continuity,  of  calculation  of  mutual  information  and  channel  ca¬ 
pacity  for  spherically  invariant  noises  and  mixtures  of  Gaussian  probabilities. 


This  i«  the  eubject  of  the  present  paper.  Spherically  invariant  distributions 
are  abbreviated  as  “SIM’s  *  and  Gaussian  mixtures  as  ‘GMM’i  *.  We  start 
by  studying  admissible  translates  of  SIM’s  and  GMM’s  for  two  reasons  :  the 
first  is  to  assess  the  difference  betwen  the  two  cases  and  the  second  is  that  the 
computation  of  channel  capacity  can  in  certain  cases  be  achieved  by  a  reduc¬ 
tion  to  a  family  of  translates  (Baker  (1070)).  The  conclusion  is  that  admissible 
translates  for  GMM’s  are  harder  to  come  by  than  for  SIM’s  :  the  mean  of  the 
Gaussian  measure  being  mixed  must  belong  to  the  range  of  the  square  root  of 
its  covariance  operator.  We  then  consider  the  equivalence  problem  for  certain 
Gaussian  measures  when  one  of  these  is  a  product  of  marginals  :  they  arise 
when  one  studies  the  equivalence  problem  for  joint  SIM’s  and  GMM’s  with 
respect  to  the  product  of  their  marginals.  It  turns  out  that  joint  measures  and 
products  of  marginals  are  most  often  orthogonal,  so  that  one  is  lead  to  be¬ 
lieve  that  joint  SIM’s  and  GMM's  must  be  orthogonal  to  the  product  of  their 
marginals.  This  is  indeed  the  case  when  the  mixing  function  is  smooth.  So  one 
is  left  with  the  problem  of  finding  an  "explanation  ”  for  this  orthogonality.  By 
restricting  attention  to  a  finite  mixture,  one  can  find  a  formula  for  the  aver¬ 
age  mutual  information  :  it  is  given  by  the  average  mutual  information  of  the 
Gaussian  measure  being  mixed  to  which  the  entropy  of  the  mixing  function  is 
added.  The  latter  can  be  made  as  large  as  one  wishes  by  having  the  discrete 
mixing  distribution  approach  a  continuous  one.  The  framework  within  which 
the  problem  is  addressed  may  be  found  in  Baker  (1078). 


3.  Notation,  definitions  and  nsofitl  ro salts 


f .1. 

H\  and  if)  are  real  and  separable  Hilbert  spaces  with  respective  inner 
11  2  2 

products  <  A},  A)  >i  and  <  Aj,  Aj  >j.  B  is  Hi  X  if)  and  has  elements 
A  =  (A1,  A3).  The  inner  product 

<£l,£)>  **  <A|,A)>i  +  <  A3,  A)  >) 

makes  if  into  a  real  and  separable  Hilbert  space.  The  Borel  sets  of  if,  H\ ,  if) 
are  connected  by  the  equality 

B[H\  m  8(/fj]  ®  8[ff)J. 

(Partharasarathy  (1067)  :  p.6,  Theorem  1.10.). 

Let  pi  and  p)  be  the  projections  of  H  with  respective  ranges  H\  x  (0)}  and 
{#l}  x  J5T),  J\  and  J)  be  the  maps  JjfA1,  A3)  as  A1  and  /)( A1,  A3)  as  A3. 
From  “first  principles*  one  has  : 

Jj  A1  as  (A1,  0)),  J)A3  »  (0\t  A2), 

Jf  =  »dffl ,  J)J)  *  »d/y,  ,  Jf  ^1  as  PI,  J)  J)  a*  P), 

J,  Jj  »  0,  j)jf  as  o,  and  pj  +  P)  -  idH. 


t.t. 

Let  P  be  aprobability  measure  on  B[£fJ.  P1  as  Pojj-1  and  P3  »  Po 
are  the  marginals  of  P,  defined  on  5[ifi]  and  B(J7)]  respectively,  and  P® 
stands  for  P1  ®  P3. 


Let  K  be  a  real  and  separable  Hilbert  (pace.  A  Gaussian  probability  measure 
Q,  defined  on  3 [If],  is  denoted  Q  ~  Nr  (mg ,  Rq). 

For  P  ~  Nff(rfip,  Rp)  write  : 

ApsJl  ApJ*,  /ip  =  Jj  Rp  Jj ,  Aps7i3pPi+P|3pP). 

Then  (Baker  (1973)  p.280  :  Proposition  2)  : 

P*  -  ^(m*p,  R}>),  Nff(Ap,  R®), 

and  £{(Rp)*/3}  £  Rp)  where  &(7*)  denotes  the  range  of  the  operator 
r.  Furthermore,  dip  €  ^{(Rp)1/3}  implies 

(amj,,  Mp)  €  a  >  0,  /9  ^  0,  ind 

if  and  only  if  mj,  €  £({(Rp)1/3})  and  nip  €  ^({(Rp)1/3}).  Indeed, 

<  (0«*p»  7«p),  £>*  <  (0  +7)*{<  nip,  k1  >1  +  <  «p,  k3  >|} 

and  <  dtp,  It  >3  <  C*  <  RpK,K  >  (Baker  (1970)  :  p.S,  Corollary  2,e)).  So, 
choosing  for  k  the  element  Jj  k1  for  example,  one  has 

<  mj>,  k1  >3<  C-  <  Rpk1,  k1  >1  . 

Finally,  <  Rp£,  £  >  =  <  Rpk*,  k*  +  <  Rpk3,  k3  >j  (Baker  (1973), 
p.280  ).  Thus  : 


<  (£mp,  7»»p),  K  >S  £  C(0  +  7)*  <  R®£,  K  >, 


which  i«  enough  to  prove  the  first  claim  (Baker  (1070)  :  p.5,  Corollary  3,e). 
The  second  is  proved  similarly.  For  example, 

<  dp,  K  >3  <  2{<  nip,  A1  >1  +  <  flip,  A3  >j) 

<  2mox{Ci,  Cj}(<  TZpA1,  A1  >1  +  <  UpA3,  A3  >j). 

£4.  Gaussian  mixture  measures  (GMM’s) 

Let  Q  ~  %(mg,  Hg)  and  Tjr^A  =  ak, k  €K,  a r  >  0.  Set  Qa  * 
and  «g(a,2})  =  Qa(B),  5  €  S(Jf]»  a  >  0.  Then,  through  the  explicit  form  of 
a  Gaussian  measure  (Parthasarathy  (1067)  :  p.170,  Theorem  4.0)  and  Fubini’s 
theorem,  one  has  that 

Qa  ~  %(omgi  oA  &q)  and  jg  is  a  transition  function  on  £R+  X  B[K\. 
Let  F  be  a  probability  on  A  GMM  is  a  probability  of  the  form  : 

roo 

Qf(B )  *  jf  7g(a,  B)F{da)y  B  €  B[K] 

2.5.  Spherically  invariant  measures  (SIM's) 

Q  and  Tj^a  are  as  in  2.4.  Furthermore  Qa  ~  ^|f(mQ*  <?Rq)  aud 
$g(a,i?)  =  Qa (B),  B  €  B[K]  ,  a  >  0.  $g  is  again  a  transition  function 
and  a  SIM  is  a  probability  of  the  form  : 

dr(B)  -  B  e  B(K]. 

2.0.  Second  order  properties  of  GMM's  and  SIM's 
Let  A  be  a  random  variable  with  law  F.  One  may  cheek  that,  provided  BA? 
exists, 

i)  Qp  has  mean  BA  •  mg  and  covariance  BA?  *  Rq  +  V A  •  mg  ®  mg, 

ii)  Qp  has  mean  mg  and  covariance  BA?  • fig . 
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Furthermore,  ^{(.Kg)1/*}  £  ^{(Bgj)1/3}  ,  with  equality  if  and  only  if 
mq  €  This  may  be  «een  u  follows.  The  square  root  of  mg®  mg 

is  {«*g /\/llrog ll Jf  > ® (mg />/ll mQ ll* }  > 80  that  £{(Bgr  )*/*}  i* the  linear 
manifold  generated  by  £{(/2g  )*/*}  and  mg  (Baker  (1970) :  p.6,  Corollary  2; 
Sytaya  (1969)  :  p.607,  Lemma  2)  . 

g.7.  GMM’t  and  SIM's  on  product  spaces 

Let  P  ~  Nff(Ap,  Rp )  and  F  be  a  probability  on  B[ZRf]  .  Pp  and  Pp 
are  defined  as  in  2.4  and  2.5  respectively.  Using  again  the  explicit  form  of  the 
characteristic  function  of  a  Gaussian  measure  (Parthusarathy  (1967)  :  p.179, 
Theorem  4.9),  one  has  that  : 

i)  (P  o  T~lay  ss  P*  o  JJJ1  the  result  begin  written  Pi, 

ii)  Pa  ®  Pa  —  P®  °  written  P®  . 

The  following  notation  shall  be  used  : 

Pi  S3  />*  O  jr1,  />®  SB  Pi®  Pi, 

R®J!  =  a*piRPn  +01nRpft. 

Then  one  has  also  : 

;■)  -  Nnttam'p,  tm*p),  R®^), 

/>)  Pfj,  ~  *ir(>*p.  **,), 

*)  P>(B.)  -  f°°  Pi(Bi)F(ia),  fl;  €  *[*,], 

JO 
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fc-  i 


**)  A>(fl.)  -  f  pi(Bi)F(da),  B;  €  B(tf,J, 

./o 

0  (*)  =  nr  ’  Pfj(B)F  ®  F(<fe»,  40),  B  €  fl(ffl, 

«)  Pp(B)  =  nr  ®  *(<<<».  <W,  B  e  B[HJ. 

f.5.  TVons/ate*  of  GMM’t  and  SIM’* 

Let  Q  ~  %(mQ,  i2g)  and  =  k  +  a,  k  €  K,  a  €  K.  Then  one  writes 

<3S  for  «  O  T^)  o  SK^  =  (<3  o  S“J,)  .  r~l0.  One  hu  : 

Qf(B)  =  (Of  o  5i‘s)(B)  =  /°°  Ql(B)F(4a),  B  6  S[JrJ. 

%/o 


V 


S.  Eqalnknei  and  singularity  r  asm  its 


SKi 


5.1.  l%e  ease  of  translates  of  GMM's  and  SIM's 
a)  Let  Q  ~  %(mg,  Ag)  and  Ag  have  the  representation 
Kg  *  E,‘ ««f ®  «i*  where  Age,-  =  p,-e,-,p,-  >  0, 
the  e,-’s  are  orthonormal  and  <  °°-  Set  : 


1>n(k)  =s  (1/n)  ^{<  *,«,•  >|f  /ft], 


^(A:)=sIimfup^n(A). 


Then, 


i)  if  mg  €  A(v^), 


Qer{*  €  K  :  jf(k)  =  a5}  *  $a{*  6  *  :  ^(*)  *  a3}  *  1, 


i 

/iV’ 

id 

*s 


ii)  if  mg  £  £(-y/Hg),  nothing  can  be  generally  asserted  about  the  behaviour 
of  rj>n. 

Indeed  : 

i)  Let  A;  =<  mg,  e,-  /y/p.  and  Xt-(k)  =<  k  -  amg,  ef-  >k  /Jp-. 

Then: 

y.„(*)  =  4(*)  +  ^(*)+^(*), 


^(*)-(i/n)  £>?(*), 

*=1 

n 

(*)  *  2a(l/n)  ^  xi*i » 

i=l 

n 

^(*)-c3(l/n)^A?. 


w 


L et  pn  =  2J?=i  ®  •  Then 


ww 

Y  A;et-  =  1  Pn  mQ  » 


so  that 


=  IKn/^q)  l9nmQ\\^. 


Since  mg  €  £(\/-Rg)i  IK-y^Q)  lmQlljf  <  oo,  so  that  A;  <  oo  and 

limn  ^»(*)  =  0. 

With  respect  to  Qpft  {y,-(fc)  =*  At-X,-(A)}  is  a  sequence  of  independent  ran¬ 
dom  variables,  each  of  which  has  mean  equal  to  zero.  Furthermore, 

oo 

VYi  =  A ?VXi  «  <*JaJ,  so  that  <  oo. 


Thus  limn  V'n(^)  =  0,  Q<*  ~  a-*.  (Neveu  (1965):  p.146,  Proposition  IV.6.1). 
Finally,  limn  4(^)  —  <**iQa  —  a.*.  by  the  law  of  large  numbers, 
ii)  Let  now  K  be  /j,?,*  be  that  element  of /j  which  has  components  equal  to 
zero,  except  for  the  i-th,  which  is  equal  to  one.  nig  has  components  mt*  and 
nig  £  *(\/Rg)  means  ]£Sl  (**?/P»)  =  oo.  Choose  for  m;  the  value  y/pf  /i. 
Then  nig  £  %{y/R g).  However, 


£  «  £(i/0  -  I«S"  +  o  +  o(i) 


1=1  »=1 


(Pisot  and  Zamansky  (1959)  :  p.523),  so  that 

limn  ^n(^)  =»  0.  Now  (VYs/i*)  a  ar*/*5,  which  has  a  convergent  sum,  and 


limn  #»(£)  =  0,Qa  -  a.t..  So,  i>(k)  =  a*,Qa  —  a.t..  Choose  now  for  m,- 
the  value  Since 

n 

38  t  *(>/*£). 

»=1 

Furthermore,  Hmn  4>n(k)  =  ar202.  Now  (VY{/«2)  =  af2£2/t2,  which  hat  ttill 
a  convergent  ram,  to  that  limn  i>n(k)  —  0-  Consequently,  j>(k)  =  ar2(l  +  02). 
Finally,  if  in,-  hat  the  value  \/ p{  •  ,  0  <  0  <  1, 

£("*i/«)=Xy  >r.*+,/(/»  +  D, 

«=I  1=1 

so  that  limn  V'n(^)  =*  But  (VYj/t*)  =  or2/i2“^,  which  it  the  term  of  a 
convergent  teriet,  to  that,  ttill,  limn  V’n  (fc)  —  0. 

Consequently,  iimn  j>n(k)  =  oo,k  €  K. 
b)  Q"F±QF  or  (JJ.  X  i)F  implies  a  * 

Indeed  : 

Let  B  €  b\K\  be  tuch  that  Qp(B)  =  Qp(Be)  =  0.  Define  : 

C  =  {a  >  0  :  Qfl,(Bc)  *  0},Co  =  {a  >  0  :  <?{*(*)  »  0}. 

C  and  Co  may  be  assumed  to  be  measurable.  Thus,  since  F(C)  =  F(Ca)  —  I, 
F(C  H  Co)  =  1  and  consequently  CnCa  ^0.  So  there  it  tome  a  for  which 
Qa  -L  Qa-  Now,  since  Qa  rsj  NK(amq,  a*Rq)  and 

Qa  ~  /Vjr(a  +  armg,  o^Rq),  one  mutt  have  a  £  Z(y/Bq)  (Rao  and 
Varadarajan  (1068) :  p.805,  Remark;  p.808,  Theorem  4.1;  p.812,  Theorem  5.1). 

<00 


a  £  *(\/Eq)  implies  OJ.  JL  Of 


a  £  l(y/Rq)  and  mg  6  *(\/j*Q)  $F  -1  Qf 


*) 

Indeed  : 

Let  Xn(k)  ®<  *>*»  >K  “d  Bn  -  {*  €  K  :  Xn(k)  >  ew}.  Then  : 
QF(Bn)  =  /0°°  Qa(Bn)F{da)  and,  with  respect  to  Q*, 

Xn  ~  JVjr(a  <  mg, An  >k  i®*  <  *«•  >lt)* 

Thus,  for  a  >  0  and  <  Rq  kn,  kn  >k  *  °»  Qa(Rn)  -  1  -  #•«)»  where  <f>  it 
the  distribution  function  of  a  random  variable  which  is  distributed  N( 0, 1),  and 
tn  =  (cn  -  at  <  mg, An  >|ir)/a|l>/^Q^>lljr>  Similarly,  Q%(Bn)  »  1  — #tn), 
where 

tn  a  (fin  “<»  <  mg,  An  >r  -  <  <*,  A»  >jr)/Qfll * 

So,  if  cn  can  be  chosen  so  that  limn  »n  =  oo  and  limn  tn  =  — oo,  one  has 
Qp  JL  Qp.  For  SIM’s,  one  need  only  drop  the  a  in  front  of  <  mg,fcn  >%. 
The  corresponding  constants  shall  be  denoted  in  and  tn . 

The  assumption  mg  €  %{y/Rq)  implies  that 

<  mg,  kn  >k  /IIv^qMIjC 

is  bounded,  and  a  £  R(y/Rq)  implies  that  kn  can  be  so  chosen  that 
<  a,  An  >k>  dn-  <  Rq  kn,  kn  >r,  with  dn  arbitrarily  large  and 
ll^nlljT  “  I  (Baker  (1970)  :  p.5,  Corollary  2,e)).  Thus,  choosing  for  GMM’s, 
«n=<»,in  >K  /a>  “d,  for  SIM’s, 


«n  m  (<  «i kn  >k  /2)+  <  mg,  kn  >jfi 


one  ha*  limn  #i»  =*  limn  l»  =»oo  and  limn  *n  =  limn  in 


— oo. 


So,  one  i*  left  to  consider  the  case*  or  a  0  and  <  Rq  hn ,  kn  >Kwm 
some  n.  Suppose  thus  that  kno  —  ho  is  such  that  <  Rq  Icq,  Icq  >jp «■  0.  Let 
X(h)  =»<  A, Icq  >f[.  Since  mq  6  &(y/Rq)  implies  <  mg,  Icq  >k=  0  (Baker 
(1970)  :  p.5,  Corollary  2,e)),  X  is,  with  respect  to  Qa*  concentrated  at  0,  and, 
for  the  same  reason,  at  <  a, Icq  >k>  0,  with  respect  to  {}£.  In  the  case  of 
SIM’s  the  respective  points  are  <  mg ,  k q  >jq  and 

<  mq,kQ  >k  +<  *,kQ  >jf .  Thu* 

Qp(X  *  0)  m  t)p(X  *■<  mg, Icq  >jf)  «  1 


and 

Qf(X  =  0)  =  Qf(X  =<  mg, Icq  >k)  «  0. 

Now,  if  or  m  0,  Xn,  with  respect  to  Qq  is  concentrated  at  0,  and,  with  respect 
to  <?q,  at  <  a,kn  >k-  Thus,  since 

<  a, kn  >k>  0,  Qo(<X»  >  cn)  =  0  and  <?QpCn  >  en)  *■  1.  In  the  SIM’s 
case,  the  values  are,  respectively, 

<  »»Q,*o  >K  <  »»g,  kn  >  +  <  a,kn  >k,  with  <  mq,  kn  > 
bounded,  and  the  same  argument  applies. 

Remark  Is  The  argument  used  fails  in  the  case  of  GMM’s  for  which  mg 
Z(y/Rq)  .  In  fact,  if  k  is  such  that  <  Rqk,k  >Ksa  0,  <  a,k  >jg>  0, 
and  mg  as  a,  let  Ta  be  the  map  7aa  =*  aa,  or  6  £R+  and  S  be  the  map 
Sa  »  ar  +  1,  a  €  131+.  Then,  (Jy(B)  —  F  oT^CS)  and 
Qp(B)  »  F  o  S~l  oT^l(B).  Thus  Qp  ±  Qp  implies  F  ±  F  o S-1.  But 
the  distribution  function  of  F  o  is  that  of  F  shifted  to  the  right  by  1, 
and  in  general,  the  supports  of  F  and  F  o  S~l  overlap.  This  also  shows  that 
Qp  a  Qp  does  not  obtain  either. 
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The  argument  used  in  proving  singularity  can  be  found  in  Pan  ((1973):  p.12) 
where  it  is  applied  to  the  Gaussian  case.  Its  application  to  SIM's  induced  on 
by  real  processes  is  to  be  found  in  Huang  and  Cambanis  ((1979)  :  Part 
3.).  The  aero  mean  SIM  case  has  been  addressed  in  Sytaya  ((1969)  :  p.508, 
Theorem  1). 

d)  If  a  6  £{(Rq)1^8}  end  F  has  no  mass  at  the  origin,  Qp  9  Qp  and 

Qf  s  Qf 

For  a  >  0,  one  has  Q%  3  Qa  and  Qa  a  Qa  (Rao  and  Varadarajan  (1968): 
p.312;  Theorem  5.1).  For  o  =  0,  Q%  is  concentrated  at  a  and  Qa  at  0,  Qp  at 
a  +  mg  and  Qp  at  mg,  so  that  the  assumption  on  F  is  required. 

e)  Qp  =  Qp  implies  a  €  ^{(Rg)1/8}  and  both  Qp  s  Qp  and 
m  €  ^{(Rg)1/8}  imply  a  €  ^{(Rg)1/8} 

f)  Using  a)  and  (Siorohod  (1974)  :  p.99,  Theorem  2),  one  has  : 
i)  if  a  and  mg  belong  to  ^{(Rg)1^8},  then 

W%/dQp]{x)  = 

n  _ 

cxp{[1>(x)]~l  Urn  Pi'  <x~  Vt(7)mQ>H  >K<  «*>«»  >K 
”  $=1 

“ll(>/Rg)  lallif/3}> 

ii)  if  furthermore  a  belongs  to  £(Rg),  then 

WfWpIW  ”  <  *  -  VSW-HJ,  Aq1*  >K 

The  formulae  for  SIM's  are  obtained  by  dropping  the  in  front  of  mg. 

Remark  2s  The  latter  can  be  found,  for  the  aero  mean  case,  in  Sytaya  ((1969): 
p.509,  Theorem  2). 


9.9.  The  mm  of  joint  measures  and  measures  which  ore 
product*  of  merginele  :  the  Geuecien  case 

The  notation  is  that  at  34,  2.4  and  24.  The  basic  assumption  is  that  P  ~ 

»)  If  |a  -  t)  >  0  or  \fi  -  <|  >  0,  P»g  ±  P»f .  CwMqaartly,  1?^  M  P® 
implies  a  =®  7  and  0  =  6 
Indeed  : 

If  Q  **  %(s»(j,  Rq)  and  |a  —  0\  >  0,  Qa  1  Qfi,  far  then 

Rq.  -  N/H^’(«ir  +  «d  T  -  [(a*/**)  -  !)«*. 

which  is  not  Hilbert-Sdunidt  (Rao  and  Varadanyan  (1068)  :  p.312,  Theorem 
5.1).  So,  in  case  |a  —  7)  >  0  but  0  =  6,  choose  B  =  Ax  £Tj,  with  A  such  that 
Pa  (A)  =*  P*(Ae)  =  0,  and  in  case  \0  —  4|  >  0  also,  choose 

B  =  (Ax  iTiM#!  XC),  with  JPj(C)  »  Pf  (Cc)  -  0 ,  to  obtain  P%j(B)  = 
P°f{B*)-0. 

b)  Let  a,  0  and  7  be  positive  and  assume  P  X  P®.  Then  Pa  X  P and 
Indeed: 

Since  all  the  measures  considered  are  Gaussian,  if  the  result  is  false,  one 
must  have,  for  example,  Pa  —  P®  ^  (Rao  and  Varadarajan  (1068)  :  p.308, 
Theorem  4.1).  The  proof  consists  in  showing  that  the  latter  implies  P  3  P®. 
Since  P  and  P®  have  the  same  mean  (2.3),  one  must  only  check  that 

R|  -  +  S)y/fy, 

S  Hilbert-Sdunidt,  self-adjoint,  and  such  that  o(S)  >  —1  (Rao  and  Varadara¬ 
jan  (1068) :  p.312,  Theorem  5.1)  :  this  condition  will  be  referred  to  as  the  *H-S 
condition*  henceforth. 


Since  Pa  ~  Nn(ai&P,  a*Rp )  and  Pj^  ~  Wff((0mp,  7 »*p)>  #fj7)  (2-4 
and  2.7)  and  since  it  is  assumed  that  P<*=  P^,  the  H-S  condition  obtains  : 


0  *ft,  -  a*{(Rp)l/,K«H  +  ^{(Rp)1^}, 

T  Hilbert-Schmidt,  self-adjoint,  <r(T)  >  —  1.  By  2.7  one  has  that 
a  0,  or,  equivalently,  using  t), 

<0  nKpn  +nURp)l/i}TWP)l/i}n  -  o. 

Similarly,  using  2.3.,  one  has  that  Rp  —  Rp  a  —  PlRpn  ~  PtR-PPh  so  that, 
from  it),  one  gets  : 

Hi)  Rf-Rp-  H{(Rp)1/,}r{(Hp)1/,}M+M<(RF)I/,}r{(«F)1/,}«. 

2.7  also  yields 

<  PlRPPl%i  S><<Rf^K>/»>, 

so  that,  by  i), 

<  p\ Rpp\ K,  K>  <  r-  <  RpK,  K >, 

for  some  appropriate  r.  This,  in  turn,  implies  (Douglas  (1966) :  p.418,  Theorem 

1): 

iv)  piRppi  a  {(Pp)1/*}CT{(Pp)1/3)>  U  :  H  -*  H  bounded,  self-adjoint 
and  >  0.  One  has  similarly  : 

v)  PIRPPI  *  {(PpJ^JV^Pp)1/*},  V  :  H  — ►  H  bounded,  self-adjoint 
and  >  0.  The  polar  decomposition  (Weidmann  (1960)  :  p.197,  Theorem  7.20) 
now  yields  : 


T>)  {(&p)ll*}f\  =  Afoflppi)1/3,  A  a  partial  isometry  with  initial  set  L\ 
which  if  the  cloture  of  £([pi  Rpvi^^2)  and  final  set  L%  which  is  the  closure 
of  *({(/*p)1/,}pi).  Similarly,  one  has  : 

»ii)  s/U{(Rp)ll')  -  ^({(^p)1^,)V{(ilp)1/*})1/*. 

However  A0 A  is  the  projection  p^v  onto  £j.  and  AA*  the  projection  p£r 
onto  I,,  wi)  thus  yields  A0{(Hp)*/2}pj  =■  (pjPppj)1/2,  and,  similarly,  vii) 
yields  P0 >^J{(Pp)1/*}  =*  It  follows  then  from 

iv)  that  A0{(«p)1/3}p1  -  ^^{(Pp)1/2},  which  can  be  rewritten  as 

Tiii)  Piffftp)1/3}  -  {(Rp)l^)s/VBA\ 

Similarly,  one  has  : 

«)  W<(«p),/3}  -  {(RP)l^WVDC’. 

So,  using  viii)  and  ix)  in  iii),  one  has,  setting  f  =*  /UBA*TCD*/V, 

x)  R$  =  {(Rp)1/2}(ids  +  T+i*){(RP)1/*} 

One  must  finally  show  that  <r(T  +  T*)  >  -1  ,  or,  equivalently,  that 

and  {(fip)1/2}  have  the  same  range.  Now  \J~R%  and  yjR®^  have  the  same 
range,  since 

0  <  .aWVljJ  < 


(Douglas  (1966)  :  p.41S,  Theorem  1).  The  H*S  condition  due  to  Pa  a 
insures  similarly  that  ay/ Up  and  \Jr®^  have  the  same  range.  Since  cty/Rp 
and  \/5p  have  the  same  range,  the  H-S  condition  on  P  and  P®  obtains, 
leading  to  a  contradiction. 

c)  Let  P  a  P®  and  assume  that  the  closures  of  ft(^/Pppi)  and  £(<V/Apps) 
have  infinite  dimension.  Then  Pa  a  P^  if  and  only  if  a  *■  fi  »  7  (thus 
otherwise  Pa  ±  P^7). 


Remark  3s  That  the  cloture  of  R{y/Rp9\)  ha#  positive  dimension  mean* 
that  the  range  of  Rp  if  not  contained  in  {0j}  x  :  indeed,  if  the  dimeneion 
if  aero,  the  kernel  of  \/Rp  contains  H\  x  {0j  },  and,  since  Rp  and  s/Rp  have 
the  same  kernel,  H\  x  {fj }  is  contained  in  the  orthogonal  to  the  closure  of 
Z(Rp),  so  that  {0i }  x  £Tj  contains  the  closure  of  &(Rp).  infinite  dimension 
is  obtained  by  choosing,  for  example,  Rp  to  be  an  injection. 

Proof  of  c) 

Assume  thus  that  Pa  a  Pf^-  The  H-S  condition  for  Pa  and  P®7  implies 
positive  constants,  I*i  and  Tj,  for  which 

ri ‘<RPa£,K>  <  £  rr <RpaH1H> 

Choosing  for  p\J?  one  gets  : 

«*r,||JipnJS,||*  <  /3,||fi/.piA’||s  <  a5r,||«/»p,^||J. 


in  particular  that  y/Rp~  and  yjR^  ^  have  identical  ranges,  so  that  there  exist 


Since  there  exists  at  least  one  h!  such  ||RpPi^||  >  0,  one  has 

aJrj  <  fP  <  ar3 Tj,  so  that  f)  ■»  0  if  and  only  if  a  *•  0.  The  same 
is  true  for  7.  One  may  thus  assume  that  a  >  0. 

Let  A®  be  the  closure  of  the  range  of  the  square  root  of  J\R^^Ji ,  L® 
be  that  of  the  range  of  yjR®  U\  be  the  partial  isometry  with  K®  as 
initial  set  and  L®  as  final  set.  Similarly,  K\  corresponds  to  J  1  Rpa  J\ ,  L\  to 
yj Rpa  J*  and  the  related  isometry  is  V\ .  The  polar  decomposition  (Weidmann 
(1080) :  p.197,  Theorem  7.20)  yields  : 
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But  a  p*JiRPJj,  so  that  -  PU\{J\RpJi)1^'  Similarly, 

y/R^J*  -  aFi(JillpJf)1/*.  Since  (Ji*p.  Jf)1/*  and  (JiApJf)1/*  hare 
the  tame  range,  (Jj-RpJj  )*/*  a  V*  y/Rpa  J\  /or,  and  ,  consequently, 

»  Wa)t/iV,*V«f74- 

Let  5j  a  (0/a)UiVf.  Then 

V^Pl  -  $1  V*paPi .  Similarly,  *  Siy/Rpan- 

Thuf  *  Si^/SJTpi  +53v//2pap3. 

The  H-S  condition  on  Pa  and  P®^  can  thus  be  rewritten,  letting 
6*  a  /93/a8  and  ca  *  7J/a5, 

*P  -  6*pi  Pppi  -  e* PiRpn  m 

(Piy/RpS*  +pjV^5j)r(5iV^Pi  +53>y/Ppp3), 

which  yields,  after  pre-and  post- multiplication  by  p\ , 

(1-  —  4*)|| V‘RpnM\*  —  <  Si-TSi  y/Rppt&-,  y/Rppxhr  >-, 
which  becomes  in  turn,  by  continuity, 

(i  -  6*)||£||3  -  <  sf  rsi£,£  >,  Keiu 

for  y/RpJ*  and  y/RpmJ\  have  the  same  range  and  J\H\  a  piH  . 

Assume  now  1  -  A*  ft  0.  Then  either  (SfrSi)1/*  or  (-5|T5i)1/s  has  the 
same  range  as  the  identity  on  L\  (Douglas  (1066)  :  p.413,  Theorem  1),  that 
is,  either  ( SfTSj )*/*  or  (-SfrSj)1/*  has  closed  range.  But  these  operators 


are  compact  and  can  only  have  closed  range  if  L\  has  finite  dimension.  Thus 
1  —  A*  =*  0,  that  is  a  »  $.  a  **  7  is  obtained  similarly. 

3.3.  The  ease  of  joint  measures  and  measures  which  are  produets 
of  marginals  :  the  ease  of  GMM'e  and  SIM's  with  smooth  mixing. 

Let  F  be  continuous  and  without  mass  at  the  origin.  Assume  further  that 
nip  6  Z(yfR$).  Then  PF  J.  P®  and  PF  ±  Pf  . 

Proof : 

By  2.3,  n»p  €  %{\J Rp)  and  mp  €  %(yjRp)  .  Define  successively, 


Rpc;  9  Rpf;  =»  Pf fif 

i;  -  -  jJ  (^rVi, 

*,■(*)  =*< »,  >,  «<  *,•  >. 

*»(*)  -  (l/»)  £>?(*),*•(*)  -  (!/„)£**(*), 

*=1  »=1 

#3)  • lim  sup  fa  (3),  lK3)  «■  Mm  sup  (3). 
a  » 

Now,  with  respect  to  Pa,  X{  has  mean  equal  to 

o  <  nip,  fa  >«<  (^^)“1nip,  C{  >|  . 

Furthermore,  cov  (X,-,  Xj)  =*  a*  <  Rpfc,fy  >■  a*  <  «j,ey  >j-»  a3A,-j. 

Consequently,  as  in  3.1, a),  ^(f)  =■  ^(f)  —  a*,  w. .  Similarly,  with  respect 
*°  ^(3)  *■  and  ^>(3)  =*  7*,  *.*.  .  Let  now  A  be  the  diagonal  in  JH+ 

and  set  B  m  (^,  ^)_1(A).  Then 

f°°  Pe,(Bc)F{da)  m  0 
JO 
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pf(B)  =  nr  Pfyl(B)F  ®  F(d0,  da)  *  F  ®  F(A). 

We  are  thus  left  to  show  that  F®F(A)  a  0.  Fix  an  integer  p  and  let  Ap  «■ 
{(x,  x)  :  0  <  x  <  p}.  Let  Bt*  be  the  box  centered  at  (|2»  +  l]/2n,  [2*  +  l)/2n) 
and  equal  to  ]•/»*,  («  +  l)/n]x]i/n,  (i  +  l)/nj.  Then  Ap  £  where 

q  =s  np  —  1.  Then 

9 

F  ®  F(Ap)  <  £{F((i  +  lj/n)  -  F(./n»J 
i=0 

<  F(p)  max  {F([»  +  lj/n)  -  Ftf/n)} 

l<t<»p 

Since  F  is  continuous,  it  is  uniformly  continuous  on  [0,  p]  and,  consequently, 


1 


m 


I 

1 

•’S 


lim  sup  max  {F([t  +  l]/r»)  -  F(«‘/n)}  =  0. 

»  l<»<«p 

Thus  F  ®  F(A)  a*  F  ®  F(UpAp)  **  0. 

$4-  cose  of  joint  measure*  and  measures  which  are  products 
of  marginals  :  the  case  of  GMM’s  and  SIM’s  with  finite  support. 

Let  0  <  orj  <  aj  <  •  •  •  <  a*  be  the  support  of  F  with  mass  p;  €j0,  l[  at 
a;,  1  <  «  <  n.  We  write  P„*  and  Pf®.  for,  respectively,  Pa<  and  . 

a)  Let  /^,*",Pm  and  <?i,  *”,<?»  be  probability  measures  on  (Q,4)  such 
that  Pf  JL  Qy,  1  <  *  <  m,  1  <  /  <  n.  Then  there  exists  a  set  A  6  d  such 
that  P;(A)  »  Qy(Ae)  »  0,  1  £  i  <  m,  1  <  ;  <  n. 

Indeed  : 

For  measures  A  ,p  and  u  such  that  X  ±  u  and  A  A.  p,  X  ±  p  +  u 
(Ash  (1072)  :  p.67,  2.2.5  Lemma).  By  induction,  one  obtains  that 

m  n 

E*  J-E«i 

<=i  y=i 


and  A  is  a  set  such  that 


(£w*(E«/W*  o. 

»=I  J=1 

b) Ps  P®  implies  Pp  «  Pp,  but  never  Pp  S  Pp  whenever  Z(y/Rpp i) 
and  %(y/ Rppi)  have  infinite  dimension. 

Indeed  : 

By  3.2,  c),  P;  ±  P^,  1.5s  »*/»*  <  r», /  5*  fc.  Choose  B  such  that,  for 
P*  C  {/*)|,***,/^»}  and 

Pf*  €  (P,>.  /  *  M  S  /,*  S  »,  }.n(B)  -  P&(BC)  -  0 

which  can  be  done  because  of  3.4, a).  Then  Pp(B)  =  0,  but 

/f(B)  **  PjPft  >  °*  Absolute  continuity  follows  from  the  assumption 

PsP®  and  2.7. 

c)  if  P  JL  P®  and  orj  >  0,  Pp  ±  Pp.  If  crj  *  0,  neither  Pjp  J.  P®,  nor 
Pf  s  Pp  obtain. 

Indeed  : 

If  aj  =  0,  Pi  is  concentrated  at  the  mean,  which  is  I,  and  similarly  for  P®|. 
So,  if  Pf(B)  =  0 ,Be  2  {*}  »d  P${Be)  >  p\  >  0. 

-1  >  0,  from  8.2  ,  b),  we  have  that  P;  6  {Pi,-  ■•,Pn}  and 
^  6  {/’?*, 1  <;,*<",} 

are  orthogonal,  so  that  the  assertion  follows  from  2. 4, a). 

d) IfPj.  J.P®,  PJ.P®. 

Indeed  : 


Pjf(B)  =  Pp{Bc)  implies  P{(B)  =  P®.(BC)  =  0  and  then  one  applies  2.7. 
e)  If  PF  «  P®  and  aj  >  0,  P  3  P®. 

Indeed  : 

U  P  j£  P®,  P  -L  P®  (Rao  and  Varadangan  (11>68)  :  p.308,  Theorem  4.1). 
But,  by  8.4, c),  we  then  have  Pp  _L  pj?.  So,  if 


Pp(B)  >  Pf(B')  =  0,  Pf(flc)  =  0 


and  Pp(H)  =  0  which  is  impossible. 

f)ir*l  >  0,  Z(y/Rppi)  and  Z(y/Rpp j)  have  infinite  dimensional  range 
and  Pp  «  Pj?,  there  exist  Bore!  sets  Bf,  1  <  *  <  n,  such  that 


[dpF/dpf)(S)  =  £(i/w){i  -  iBl(h))idp;/dP%\(h). 
i=l 

Indeed  : 

For  each  fixed  t,  !<*<«,  choose  a  Borel  set  Bf  such  that 


P;(Bi)  =  P,®  (B?)  =  0,  1  <  *  <  n,  (;,  *)  /  (t,  i). 


This  is  possible  because  of  3.4,d),  3.2, c)  and  3.4, a)  successively.  From  2.7,  we 
furthermore  have 

P,*  —  Pf® ,  1  <  *  <  n.  Let  thus 

n 

*= 1 


Then,  for  B  €  3[f/], 


But 

f  a<£p£-  -  £>/»>  [ 

Now,  P?.(£jp  s=  0,  except  possibly  when  »  —  /  =  k.  Thus,  for  *  £  j , 

Jb  =  0- 

Furthermore, 

f  ndP%  =  (i/ti)  f  | dPi/iP%]$)p?j{dR) 

Jb  JBnBf 

-  (1  /«)/>,•(*»  fl  B;)  =  (1/W)P,-(B). 

Consequently, 


-*>(*)• 

»=  1 


Remark  4:  The  case  of  and  /*®  is  treated  in  the  exact  same  way  so  that 
in  a)-f),  one  may  replace  Pp  and  P®  by,  respectively,  Pp  and  P® . 


4.  Avtr&gt  ntul  Information 

4.1.  Calculation  of  mutual  information  in  the  case  of  $4 
If  Q  is  any  probability  measure  on  £[£FJ,  Q®  the  product  of  the  marginals 
of  Q,  one  has  that  the  average  mutual  information  of  Q  is  given,  provided 
Q  «  Q® ,  by  the  formula  I(Q)  =  fg  log[dQ/dQ®]dQ.  The  entropy  of  F  is 
the  quantity  H(F)  =  —  £?.j  pf logp(*.  One  has  : 

under  the  assumptions  of  3.4/),  and  the  assumption  yitp  €  Z(y/Rp), 

i)  I(PF)  -  /(/>)  +  H(F) 

•'0  1(Pf)  -  HP)  +  B(F) 

Indeed  : 

Let  be  the  eigenvector  of  Rp  associated  with  the  eigenvalue  pf.  Let  also 

n 

=  (l/n)^2(<  k,if  >  fy/prf,  if(H)  =  limsup^n(^) 

and  Af  **  ^-1(or?)f  1  <  »  <  n.  As  in  3.1, a) 

P{(Aj)  m  1,  Pf(Aj)  =  0,  t  ft 1  <  »,  /  <  n. 

One  may  thus  assume  that  dPf/dPf*.  is  zero  outside  of  Af.  Consequently, 

*  r 

fH  \ot{iPrliP$]iPF  =  J2  n  fB  iog|  dPF/dPp)dPi, 

Jh 

--log  p,-  +  /(P*). 
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But,  because  of  Lemma  4  in  (Baker  (1078)  :  p.76),  /(Pt*)  —  /(P).  Consequently 

n 

i(pF) = £«<-■<>*« +i(p)}  =  hp)+h(f). 

i=l 

The  case  of  a  more  general  mixing  function 

Every  measure  on  the  Borel  sets  of  a  separable  metric  space  can  be  weakly 
approximated  by  a  measure  whose  support  is  a  finite  number  of  points  (Part ha¬ 
ras  arathy  (1067)  :  p.44,  Theorem  6.8).  One  may  thus  try  to  use  4.1  to  obtain 
bounds  for  the  case  of  a  mixing  F  which  does  not  have  finite  support. 

i)  Let  B  =  {k  €  H  :  \\h  —  £]|  <  a}  and  /  *  {t :  th  €  B },  K  €  B.  Then  I  is 
an  open  interval. 

Indeed  : 

Suppose  *i  €  /,  t2  €  /  and  tj  <  t  <  tj.  Let  A  =  (t2  -  t)/(t7  -  tj)  and 
1  —  A  as  (t  —  fi)(fj  —  fj).  Then  t—  A*i  +  (1  —  A)fj  and 

II'* -ill  -  IP'l  +  (1  -  A)t,|*  -  all  <  A||t,*  -  J||  +  (1  -  A)||t,*  -  ail  <  a, 

so  that  /  is  an  interval.  Let  now  t  €  /  and  =s  a  -  ||t£  —  £||.  Then,  for 

*  <  0/11*11. lie ± •)* -all  <  111* -  ail  +  «||*||  <  a, 

so  that  /  is  open  and  contains  points  other  that  1. 

b)  Suppose  Fn  converges  weakly  to  F,  that  qF,  (B)  -  /”  QF(B)Fn(da) 

and  that  Qp(B)  =  f£°  Qa(B)F(da),  B  6  b[K\.  Then  Qpm  converges  weakly 
to  Qp  . 

Indeed  : 

Let 

9(h)  *  /  Ib  (ah)F(da) 

Jo 


and 


Then 


and 


gn(R)-  f  IB(aK)Fn(da). 
JO 

QfW  -  f  9(£)P(d£) 

Jk 

Qpn  ~  j 


Let  B  be  open  :  it  is  a  union  of  open  balls,  so  that,  IB(atk),  as  a  function 
of  a,  is  the  indicator  of  an  open  set  on  the  real  line.  Consequently,  by  weak 
convergence 

g(h)  <  liminf  y»(/i), 

fl 

and  thus,  by  Fatou’s  lemma, 


QF(B)  <lim  inf  QF(B). 

n  ■ 

c)  Let  the  assumptions  of  4.1  hold  for  P  and  Fn,  where  Fn  is  a  sequence  of 
probabilities  with  finite  support  converging  to  F  weakly. 

Then 

I(PF)  <  lim  inf  H (Fn)  +  /(P), 

>1 

since  /  is  lower  semi-continuous  for  weak  convergence  (Bretagnoile  (1070)  : 
p.36)  2.3). 
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